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The reliability, factor structure, and discriminatory 
p^wer of faculty perceptions of theperformance of departmental 
admf r.istrators were investigated, usin<T the administrator Evaluation 
Survey raES) developed at the Oniversity of" Illinois* Faculty from a 
o»1br research universityin /the midwest, a smaller state university 
in '•he midwest^ and a predbminant ly black state_univeriity located in 
*he Southeast participated^ The approximately 350 factil^^^ 
represen*-ed 16 dep^rtm^ items of the 

ass were determtiaeS^f or each university^ Two functions appaared to 
-separate the three universities in terms^of. fjl the head's 
departmental lead^rship-^communicatina departmental needs to higher 
level administrators, a^l<5cating resources among faculty, directing 
prbmotibs and' tenure process, and making, difficult decisions: and (2) 
the head's departmental management--*handling routine duties, 
resolviha tensions*' encburaaing gbbd teaching, making dif f icult . 
decisions? and. improving schblarly reputation* Tb ieter«ins whether 
the aES was aMe to detect differences ^aibng de^ a 
g^ven institution,* similar multivariate* analyses w^ on_^ 
the ma ior research university's ratings. Three functions appeared to; 
separate, the department heads in •♦^erms of the department head's 
contribute on toward imprpyina the reputation* of the department, the 
head's adsiaistratlve ability, and_the head's handling_of 
curricular/lnstructibnai matters.- a reyisior of^the AES to 11 items 
was a1:?5b tested^ Overall, it is concluded that factor analysis of the 
aES sugcests tha"^ department adiih^stratbrs are. e^yaluated-by their 
faculty alona^qne g^ br factbr. Pbssible.reai^bns^for 

the ob.serve5_nntdimensi of the aES are considered, a sample 

auestipnnaire is included. (SH) 
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* Peprbductiors supplied, b^ ^r^'S are the best that can be made 

* from the or iginai document. 
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Abstract 



This study examines the usefulness of- faculty ratings as a measure 
administrator quality- Approximately 350 faculty from 16 departments 
across three universities responded to a survey which assessed t\x6it\ per- 
ceptions of the performance of their department heads- The study 
explores the reliability and validity of faculty ratings of departmental 
administtators and discusses the utility and applications' of faculty 
: ratiiigs as an integral part of formal administrator review programs in . 
a variety of institutional settings^ ; . ; 



• * * * FACULTY RATINGS AS A MEASURE OF ADMiNISTRATOR tJUALITY - 

• . ' _ v: . i 

;ln response to increased fiscal and "accountability" pressures?' colleges 
aiSi^. universities are begirihing to pay clpser attention to the performance^ of 
. dep^^mefital administrators. Many universities j notably the University of . 
iili^S, th^ State Un^ijf^rsity 'of ''New Yorlc, and Te^as Christian Oniversity , . . 
have^Etitiated fdtlnal evaluation proc-edures of ^department heads/chairpersons • 
'Other uiTiversities (e.g- Kansas State University and Ball State University)- 
while not requiring the evaluation of administrators, have developed evaluation 
procedures for department heads who wish to be evalna^ted. 

One reason why departjneot* heads are becoming the-f ocus of attetition is 
because of their importance within the urTiversity structure (Dressel and 
Reichard, 197(3; Faricy, 1974; Fishe^^ 1978; Smart ^nd Montgomery, '1976-). It 
is commonly assumed that by improving , the quality of the department hea3, the 

quality of the* department ^ college or university would also be improved (Hoyt 

^ -— - 

^ and Spangler, 1979)- This symbiotic relationship between the head and the 



department, however, has seldom been emptrtcally demonstrated (HengstleE, 

Brandenburg, Braskamp, and Smock, in press)- • 

Various criteria and appraisal methods (Farmer, 1979; Fisher, 1978; 

Genova, Ma?doff, Chin and Thomas, 1975; Smart and Mbntgbmer77 1976; and 

Shtogren, 1978) have been suggested for use in the evaluation of academic 

administrators.. Berquist and tenbrink (1978), in a survey of four hundred 

American colleges and universities, identified 'six major procedures usied in 

university, college, and departmental administrator evaluation which includes 

unstructured narration or essay appraisal, unstructured documentation, 

» * * 

structured narration, rating^ scales, structured documentation, and management 

by objectives- In the past, the unstfuttured narration or essay appraisal 
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No delineation was, made between department heads and chairpersons in this study 



combined wi^tli the confidential interview witii the dean was the most conffnon 

-9 - - - - . ^ t 

and preferred method of department head evaluation ^Lahti , 1978; Farmery I979)i 

Increasingly, faculty perceptions' and joadgmertts collected systematically are 

being conrsidered in ah evaluation _bf the department head (Sitic^rt -and Montgomery, _ 

• _ . 

1976; Fisher, 1978; Ehrle^ 1975; and Hiliway, 19733- ; . . ' 

In his review of administrator evaluation efforts, Farmer (1976) identifinsd 

three primary reasons given fi>r faculty evaluation of the department head/ 

^ — . * * • ' 

chairperson. The most common reason was related to institutional self-evaluation 

As he pointed out: ''Much of the current emphasis on the evaluation anS develop- 
ment of academic administf ators in fact brigihat.es from the trend toward: 
institutional evaluation which of necessity 'includ'es an assessment of all groups 
in the educational enterpriser" Fanner's second rationale for faculty evaluation 

of. administrators stemmed from research or immediate experience with student 

- - - • - — - - ✓ 

-.evaluations of instruction. Often^ the pressure for systematic administrator- 
devaluation caine from faculty yho developed negative attitudes toward the 

formalized pVogram of student evaluation of instruction on their campus 

- _ c ■ ■ _ 

(Peterson, 1976). The final reason was ^related to the head/chairpersbri' s ^ 
desire tb have faculty bpiniohs and judgments about their perfbrmahce for 
self -improvement purposes. 

With the increased movement toward accountability and general acceptance 

. • _ _• _ _ _ • . _ 

of the concept .of administrator evaluation, it appears that the evaluation of 

___ _ _ _ _ )__ 

department heads will cbntiriue tb increase bver the next decade- One source 

of information that will be particularly important in the evaluation will be . 

faculty ratings of the head's performance. / * _ 

A review of the major faculty rating forms of administrator performance 

reveals that all of the instruments have been developed since 1970-iL5^irlei 1975; 

Fehker, 1975; Galina, 1978; Goodwin afi^ Smich, 1979; Hiliway, 1973; La)iti, 197§; 



... • . . . . . ^ 

McCarthys- NbLe 1; Ryarij Abbot, Cook, Dcniiam., Kiniljnll, Kliiiii, and Mijl calf, 

' N6te 2:-Uhl 6t Pratt, Note 3). Typically^ the number of items in the quest ibhhaires 

ranged ^rom twenty- five (Hillway, 19735 to sixty-eight Q-QcCarfhy, -Note 1)^ and 

I most quest iorihaires incorporated' a fiive-point response scaiei 

; In comparison to student ratings of instruct idrt'^^^esearch on the 

* reliability and validity of faculty ratings of admin is t|ra tor performance has 



) • identified 



been relatively limited^ For the most||part, research, hM^oncentrated on 
identifying the underlying factor structure of the forms «sing principal 
components ^nal^sis with varimax rotation. ^ . 

'* Studies by both Fenker (1975) and Klein and Denham (197^ 
four underlying factors to their respective surveys./ In^^tite Fenker (1975) 
study the four factors were: (l) Infbrmatidn/Communication - ifelat^d to tlie 
accumulation' of pert inent info: mat io/i before acting or cbmmuniGating importknt 
information; (2) Goal Completion - related to planning activities, initiating 
and sustaining action toward a goal; (3) A delegation of responsibility factor; 
and (4) Personal Skills -'related to ease in establishing rapport, success in 

r 

working with coSn-ittees , and sensitivity to the contributions of others. 

V 

. Perhaps the most extensively used and researched administrator evaluation 

survey is the DECA system (Departmental ^Evaluation of Chairperson Activities), 
ft * • ^ 

developed by Don Hbyt and his associates at Kansas State University. The goal 

^ of the DECA system is to identify discrepancies 'between chairperson and faculty 

• ' _ ' _ ' ^ 

objectives and faculty perceptions of whetfier the chairperson is achie^ing^ 

those objectives. • • • 

• ; 

The DECA system initially grew out of a -dissertat ion by McCarthy (Note 1) 

Vho expanded Siever^s work (Note 4) on administrator' evaluation. Spangler (Note 5) 

revised McCarthy's form to 15 activity items and 33 beh'avlor items. Using data 

from *il3 departments from four public universities, split-half and intraclass 
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reliability coefficienLs were computed on the ratings of Lhe head/ciiairpcrsbn's: 
performance. Rdr the split-half reliability estimates of the activity items^ 
th|/cbg^f f icients ranged f rbm .50 -to .91 with a mediah of .81. " Fbr the intra- 
class reliability estimates, the median was i 70 and ranged fro ^49 to i8ii - 
Reliabilities for the thirty-three beha\u.or items ranged from • • — 

(split-half ireliability estimates) and from .55 to .75 (intraclass reliability 



estimates). Lin (Note 6)^ using Hillway*s (l973) initial stxrvey, found similar 
cest-retest reliability coef f icientsi Coefficient alpha for the Hillway survey 
was found to be •97. ^ ' ' - 

Spangler (jiiitc 5) also conducted several principal component factor analyses 
with vaximax and oblique rotations on the ratings of the department head/chair- 
person's performance for both the activity and behavioral items. The factor 
analysis of the averag^e faculty performance ratings (i.e., the department as 
the 'Unit of analysis) on the activity- items yielded three factors which accounted 



for seventy-four percent of the total variances The first factor was labeled 
"Personal Management Activities" arid^ included such items as rewards faculty 
appropriately^ maintains faculty morale, allocates faculty responsibilities, 
fosters faculty development and guides faculty evaluation procedures. The 

second factor, "Planning and Development," incitfded such" items as ^guides 

- - • ^ * \ _ * - 

curriculum development,- guides prgariiiatidn arid piaririirig^ fosters good teachirig, 

encourages balance among specializations , ^and faculty recruitment. The third 

factor, "Concern for Department's Reputation," centered on items dealing with 

the head/ chairperson's facilitation of excramural funding, cotranunication of 

departmental needs, improving departmerital image, arid stimullftirig research/ 

scholarly activity. ' ' 

A factor analysis of the individual faculty ratings of adrhiriistratbrs'was als 

conducted on the activity- itemsi K'ere^ only one factor was extracted^ Ail 



IS had high loadings bri. this factor ^ which strongly suggests that-the faculty 

"t '.- .__ .. ... 

ratings of the head's performance were susceptible to halo effects (Spangler, 

(Note 5). . -> ' . _ ; • . • 

* At;' is apparent ^ no consistent factor structure has emerged iq, regards to 
faculty rajfcihgs^of^adnf^ performance. Although the test-retest ; and 

internal consistentcy measures of faculty rating instruments appear : to be 
^f^irly high, generalizations are' extremely dangerous given the limited number 

of studies. . ^ • ' * 

The purpQse of the present study was thus^ to investigate the reliability, 
factor structure, and "discriminatory power of . faculty perceptions of the per- 
formance of departmental aditlinistratdrsi • 



— » METliGD 

instrume'ifit 



The instrument used •ih^this study was. the Administrator Evaluation Survey 
(AES) developed at the University of Illinbis (Office of Instructional Resources 
and office of Planning and Evaluation, Note 73i The development of tJie.AES was 
guided by a principle that the ihstxumeht should consist of a core set of items 
and an optional set to be selected by the individual" department h«Vad from an ^ 
i.tem catalog. 'Based. on a review of the literature and related questionnaires, 
nine major components of departmental admin is trat ion were identified (i.e. Faculty: 
recruitment, promotion/tenure/ salary , development; Curriculum and Instruction, Ex- 
ternal Relations': cbilege/ufiiversity , discipline, public and private agencies; 
Personnel/Admiivistratibn^ Financial Management, St^udents, Governance, Goals/Policies/ 
Prog rams /Planning J and Leadership: interpersonal, informational, and decision 
xbles5.^ Items were constructed for each component and organized into the AES 
Item Catalog. After the item catalog was completed, one or more items from 
•each component were selected for inclusion in the AE^ (Table l)* The response 



V-/ 



Tabte^l - j 



*Admintstrator Eva:luatidn Survey (AES) 



• ^ : No ^ Not / 

Poor Excellent • Opinion Applicab 



The. ie^ership of the head in * 
'the, promotion and tenure pr pees si 1 2 3 4 5 



,2. Encburagemeht of the Vchplarly and 
professional grbwthof the faculty- 



3> Facilitates appropriate balance 
among academic specializations 
■ within the department. 

4. Leadership in planning and develop- 
ing curricuiar/ instructional 
programs i • , ' 

■ ■ • 

9i EnCburageraent of good teaching 
in the department. 



6, Effectiveness in communicating 
departmental needs to the Dean 
and/or central administration^ 1 2 



mental matters i 



ERIC - ^ ■ 10 



7. Contribution toward itnprbving/ 

maintaining the reputatibh bf the 
department within th^ discipline. 



8i Support of faculty efforts to obtain 
grants and contracts from external 
sburces for faculty research. 1 2 

_. _ _ -*_./'__ 
9. Perfbrmahce in handling tHe rbutihe 

administrative affairs of the 
^ departmenti 1 2 

lb. Effectiveness in allocating - -/ 

available funds and other resoi|rces 
ambrig- the faculty^ - \ 1 , 

^ ^ . 

11. Receptivity tb faculty suggestibhs 

and opinions on important deparc- i > ' * C' 



4.-5' ■• . \ 



12m Effectiveness In providing 
academic direction to the 
depertmenti 



Table 1 (C6ht*d5 

Poor Excellent 
1 2 3-4 5i 



No Not 
iDpinion Appii sable 



13. ■ Effectiveness in revolving 

tensions within the departmeht- 



12 3 4 5 



14i Recognition and reward of faculty 
contributions to the departments % 



1 2 3 4 5 



Effectiveness in keeping^ faculty 
informed on matters of- potential 
importance or - interest • 



1 2 3 4 ^5 



16. Willingness to make difficult 
decisions- ■ - 



1 2 3 4 5 



i7i The scholarly repi?tati6n of the 
head within the dree ip tine- 



1 2 3 4 5 
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■'.£5^ihat for Uie 'l7 AES items cei^sisted 'of a Eive-pbiht bip61al^ (pbdr-cxcellent) 

__ . . . / . \ , ' 

: responSE^ scale. 

/ 

Subjects 



tin] 



The data source \itiiized in it/lis stud'y was faculty ftom three diverse . 
liversities (Universities A, B, C) whb cpmpleted the AES. University A; Is 
a major research university loca'ted in the midwest. Six departments Uom a 
s variety of disciplines (e.g., business, education, ^i^e and applied arts) 
; agreed to participate- In the study- A total of 233 faculty or 79 percent of those 
receiving the AES at University A completed the survey. The return rate of 
•the faculty in the six departments ranged from 70 to 94 percent. 

The second university participating in this .study was also- a public 
university located in .the midwest having a much smaller enrollment than 
University A. Four departments administered the AES to their faculty. The 
overall response rate from University B was 86 percent (N=49); the lowest 
return rate was 64 percent from Department Onei ; 

The third university (University C) participating in the study was a • 
predominantly black public university located in the southeast. ' Six depart- 
: ments administered the AES to their f aculty^--Th^ overall response rate from 
University G was 62 percent (p54^'^The lowest return rate was from Department. 
•Three with 43 perceni'i^''^e lower rate for University C may be due in pari to 
the fact ^hatT the evaluations of the departtrent heads were not mandatory was 
.-the case for University B and for four bf>^ the six departments at University Ai 

RESULTS f ^ ' ' . 

Factor Anal^y^4s^ • I • ^ 

; ' An exploratory factor analysis was first performed separately on the 

faculty ra'tings of the aSeS for each of the universities.^ The, extracdTon _ 

«*■ * _ ' _____ ^ 

progrars-^was ?A2, part of the SPSS statistical package (Nie, Hull, Jenkins, / 

• Steiribrenner, and B^ntj 19755. Cbmmuhalities .were estimated using squared 

* '/•■« _ I> 

^ multiple correlations, and fact6ri weri rotated to .obtain an oblique simple 

structure factor pat*terh» z ■ Q • 



' . ■ ' " • _ - ■ '•• 

-Based on thd Scree; test^ (Catell, 1966), one factor was extracted from 

!>_'__. 

yniverstty A ahd' C ratings oh the AES. Tiie percent 'of variance accounted for " 
by th^ factor was i4% and 767* for University A and .C, respectively. Fo^ 

• • y \ _j L \_ 

University however, four factors were ^extracted which accounted for 74% 

of *;the total! variance.- ;fhe first eight eigenvalues .for University B ratings ^ 
'-were: S.36,;i.74, 1.38, 1.59, vSZj; .71^ .59, .43. The first eigenvalue for 
' each university was over- f.60 suggesting a' strong and important first factor. 

The dissimilarity of the extracted factors for University B data may 'be a 

function of t^ie restricted sample size .CN=^-9).t 

Presented in Table 2 are the factor loadings (item correlations. with first 

principal axis) for University -ii and well as the factor loading for 

University B. ' As bnlicated, the Ibadirigs on Factor 1- ranged from .51 (Item 1) 

• _ _ ' 

to •71 (item I2r) for University. A and from" .78 titem 9). to .95 (Item 17)^fo? 

I ^ , — > ' 

University C. This factor for University A and was thus labeled "Overall 

. Impression of the Administrator's Performance". . " ^ 

^ ^ > * ; . % 

The initial AES factor extractred^f rom University B ratings was labeled / 



•"Administrative Leadership". Ten of the;17 AES^<L^s had loadings greater than 

^ _ _ '_ 

..30. on thii factors. The ^n items dealt ■ primarily with th-e- department head's 

• effectiveness in communicating information, making decisions, and performing * 

4 ' • ' - - ' - ^ 

admir^istrative duties. The second factor was concerned with the^ "^department 

Read»s academic leadership", ^ive items helping to define this - factor were 
" those dealing with the heads: facilitation of balance among the specializations ^ 

within the department encouragement of good teaching, receptivity to -faculty 
"^^pinions on departmental matters, planning and developing curricular/instfuc- • 

tional programs and allocation ^of funds and* resources among the faculty. The 

third "factor was 'related to' the head's "scholarly reputation", and the fourth, . 

factor involved the head's "leadership in improving the department's reputation". 



TaMe 2 ^ ; ' ' 

Faduit^ Loadings "pn Itdms on the Administratbr EvaluaLidri Survey (AES) 

Uhiversi^ 



. ITE M 



A .e B - — 

Factor 

_4 -J II 111 IV 



1 Leadership in the promotion and tenure 
process 

2 Encouragement of professional growth of 
faculty 



.51 .90 • .17 .07 .34 \il8 
.69 .90 ^49 .13 .26 : .38 



3 Facilitates balance among academic speci^li- 

zat-ion in deDaftment • ^. -65 .81 -.04 .88 -.05 



7 eontribution toward improving reputatidti 
of department , . . 

8 Support of faculty efforts to obtain 
grants from external sources 

9 Performance in handling routine admini- 
strative affairs of department' 



17 Schblarly ireputatlon of the head ' 
within *th^ discipline 



.25 



zat'ibn in department 

4 , Leadership in planning ciirricular/ instruc- 

tional programs » - ■ * * , 

5 Encourage good teaching in department 
• _ . ■ - ' ^ . - - . . • 

6 Effectiveness in communicating depti , _ ^ __ ^ ...^ 
needs to central administration . .56 .89 .91 -109 -.01 -.01 



-62 0 .04 .64- .05 -.27 
•^6' . .86 -.B5 .84 .11 -.34 



.70 .89 .34 ' .12 .36 '-•55 
.54 JiU~ .57 -.07 .05 -.03 



.62 ^78 .57 . .la .00 -.07 



id Effectiveness in allocating available _ 

funds/resources among faculty -55 ^ .85 .24 ^62 .03 .24, 

11 Receptivity. to faculty suggestions . __: 

V and opinions / -71 .88 .16^ .78 .05 .0^ 

n ^\Effcctivencss i^n providing academic , ' • -7. m 00 

•direction to department • ^. ^ ' -73 .92 .71 -.18 .01 .22 

13 EffWtivgness in resolving tensions . - 
withlnHthe department " - .69 .88 .54 .27 .02 .14, 

14 Recognition and reward of faculty con- 
tributions to the department : 

15 Effectiveness in keeping faculty informed 
* on matters of importance » • . 



.54 > .88 .74 -.07 .10 .22 



.68 ;-i82 .84 ..08 -.03 -.18 
* 16 Willingness to make difficult deliisibns .73* .81 .79 il3 -.06 -.23 



.65 - .95 -.13 --.02 .lid2 -^09 



- H - . . ^ 

The correlations among the four AES factors ranged from .55 bitwieh Factors 1 and 

: - _ - - - _ -- _ • - — _ _ - - 

2 to -•03 between factors 3 and 5. The median correlation was •19. The 

' / ■ ^ ' 

determinants of the .cbrrelatibh matrices for eafch of the universities were very 

small (i.e. less than Given the size of the determinants and the 

/ 

_\ _ _v ___ _ /_ 

» restricted sample sizes for University B and C, caution must be emphasized in 

_ y' 

making generalizations from the l^tX.6t patterns. 

. • . . ' ■ ■ <>• 

To determine the reliability of the AES, coefficient alphas (Cronbach, 1951) 

were computed separately for each University. Coefficient, alpha (U ) is an 

estimate of the internal consistency of the survey and provides a lower bound' 

on the reliability of the scale (Lord and Novick, 1968).^ Coefficient alphas 

' • 

for the AES were i96, .92, and .96 for Universities A, B, and C^ respectively^ 
indicating a relatively high degree of internal consistency. 
Discmmihaht Analysis 

One characteristic of a valid rating instrument is its ability to detect 
differences. across subgroups (i.e. its ability to discriminate). In order to 

- "_ ---- ' J 

determine whether there were significant differences in the AES among universities, 
a multivariate analysis was conducted. Using the 17 AES items as dependent 
variables, the t^ree universities wijre compared using Wilks lambda (X) criterion 
(TatrSuoka, t97t)i The Wilks* lambda statistic' was computed and found to be • 
.917 with a significance of p < .05. It was concluded that significant differences 
existed across' t;he tirree universities. 

A discriminant analysi-s was then conducted- to determine where the differences 
existed. Using Bartlett*s test for the significance of residual rdbtSi two 
signif icantxdiscriminant functions (p <*05) were identified. Ah examination 



of the total discriminatory power of the two discriminant functions (Sachdeva, 1973) 

• •v 2_ — ___ 

indicated that 23 percent (to =.23) of the variability in the discriminant space 

* _ .__ ^ _ _ . > 

was attributable to grpajp differences^ i 



Presented in Table 3 are the stahdatdii^ed discriminant wcigilLs fdr each 
of the two functions. As indicated, the two functions appeared to separate 
the three universities in terms of (i3 Head's Departmental Leadership - 
CQTTimunicatihg departmental needs to higher level administrators, allocating 
resources among faculty, directing prbmbtibh and tenure process^ and making* ; 
difficult decisions; and (25 Head's Departmental Management - handling routine 
duties, resolving tensions, encouraging good teaching, making difficult decisions, 

and improving scholarly reputation* Ten of the 17 AES items .were particularly 

__ _ i 

effective in separating the thjree universities. 

' To -determine whether the AES was able to detect differences among departments . 

within a given institution^ similar multivariate analyse's were conducted on 

University A ratings. The small number of faculty (N=5) in Selected departmejlts- 

^ _ _ ._ f ~* ' _L_ __ 

at University B and C prohibited the multivariate analyses for these respective 

universities. . - ^ • • 

, ' f ______ 

Significant differences ( X =^.293; p< .001) were also found in the ratings ^ 



"aipong the six University* A departments.^ when the 17 AES items were used, as 
cjependont variabtesi Ag^in, a discriminant analysis was conducted to determine, "^j.' .^ 
where the differences existed. * - : ^ - . 

The total discriminatory power of the set of discriminant functions wa^ - 

; ^ .... ? ^ 

2- — - -- _- _ . 

high ( ti> =.70). 'Approximately 87 percent of the total discriminatory power 

was attributable to the first three discriminant functions. Little was added 

by the. remaining two functions. Consistent with the finding, Bartlett's test 

for the significance of the residual roots indicated that only the first three 

\ • ■ 

functions were significant (p"^ -Ol) . 

The standardized discriminant weights for the three significant AES ' 

functions are presented in Table 4i A^ indicatc^d, the three functions appeared^ 

to separate the six departtt^nt heads in ternra of (1) the department head's 

^^^^ s _ _ > 



, ,AES Standardized 1' scrimlriarit WeigUts 
^ f.br Ratings from tJniversity A, B, & C 



AES item. 



P v scrittilnant Exirictton 
1 2 , 



1 Loadership in tHp promotion and tenure ^ 
process ♦ 

1 EncburnR'cment of professional growth of 
,^ f acu 1 ty 

. \ ' _ _ " • . ^ 

3 Facilitates balance ^ among academic speciali- 

'zation in department 

4 ' Leadership ^n planning curricularV instruc- 

tional programs 

* 5 Encourage good teachij-ng in department . 

6 JEf fectiveness in conuiiuhicatirig dept. 
needs to central admiriistrat ion 

_ _ _ 

7 Contrib^tibri toward improving reputation 
of department 

8 Support ofvfacalty cff^ts to obtain 
% }<trants f romXextcrnai sources 



'9 Performance in .handling routine admini- 
^ strati ve affairs of department 

lb Effectiveness in allocating availabli^ ^ 
' funds/resourtcs among 'faculty 

n Reteptivity to faculty suggestions , 
•and opinions 

12: Effectiveness in providing ac>.<iemic 
•direction to department , ; 

13 Effectiveness in resolving tensions ' 
within the department . "V. 

14^' Recognition an<t reward of faculty con- 
tributions to the department 

15 Effectiveness in keeping faculty informed 
on tiKitters of importance . 

^16 Willingness to make dif f icult.decisiohs^ 

it Scholarly rcputatich. of the head ^ 
' ' witliiS the discipline 

University A 
O . University B - • - . ; 

ERJC University C 



-i34. 

-.15 

-.06 
..12 
.28 

-.39 
-.15 . 
-._28 
-.Di 

.02 

.09- 

.11 

.31 



-.08 

;dl - 

..05 
-.31 
.46 . 

.31 

.27. 

.08 

-•71 

-.03 . 

-.25 

.14 

.69 

■ .28 

-.30 
r.45 



-.09 --^^ 

University Cer 



1 >^ 



.26 

il6 
;95 



1.00 
-.70 
.23 



. ; ;....Me.r_ ■ , 

AES STANDARDIZED DikRiHmNrREIGHTS..__ 
*FOR RATING FROM SIX UNIVERSITY A DEPARTMENTS ; 



■ ' • . DISCRIMINANT FUNCTION 

. MNteTepTOTii SPEHTEM ; / ' ' " ^ ' ^ 3 



1 . 


Leadership iii' the protnotion and tenar.e process . 


~ no 


-.03 

0 


/ 


2 


• - 

EncouragemeiiC of professional growth of faculty 
' * .. 








•3 


Facilitates balance among academic specialization in department 




.0/ ' 




4 


teadership in planning curricular/ins true tional programs " 


,01 


.26 


.88 


5 


Encouragement of good teaching in department • ^ . 


.'20 


-.26 


. -•29 


6- 


Effectiveness iii coradiunicating department" needs to central administration 


-.21 

> 


-.10 




7 

'» 


Contribution toward iinprbvihg reputation of department 




.11 


; -.56 


8 


Support of faculty efforts. to obtain grants from external sources 


" .32 


-.27 


.06 


* 

9 


PerforniancS in handling routine administrative affairs of department ^ ^ 


-.32 


.73 


1- 

-.5/ 


10 


Effectiveness in allocatnig available funds/resources among faculty 




.il 


A 

.LL . 


11 


Receptivity to tacuity suggestions ana opinions 


.12 


-.19 


• .22 • 


• 

12 


Effectiveness in providing academic direction to the department 


..56 




-.51' 


13 


Effectiveness in resolving tensions withi^i the department • ' 


-.21 




-.23 


14 


Recdgnitiiin and reward of faculty contributions to the deiJartmeht 


.34 




;2e ' 


15- 


■.Effectiveness in keeping faculty inforned on matters of importance 

^ c 


-.42 


. .05' 


-.26 ''^ 


ir 


Willingness to make "difficult decisions ' • v 


• .19 


.73 


.88 ; 












17 


• Scholarly •■reputation ot the head within thHiscip^ ' 


• 


-.70 


,26 



%ihU ir (Cbfit'd) 



DEPARTMENTAL CENTROIDS 



Department 1 
Department 2 
^iepartment 3 
Department 4 
Department 5 
Department 6 . 



f. 





. -2- 




-2.84 


* 

1.01 


- .00 


-3.95 


.85 . 


- .45 


-2; 3D 


.73 


.36 


.3.3P 


.74 


1.18 


^.30 " 


-1.25 


,90 


-4,00 


' 1.70 


.80 
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contribution towards improving the repcitatioh- of the d^spantoneat;. (2) the 
head's administrative ability; and (3) the head's handling of curricuiar/ 

- • ' . _ ' .... : - - . ^. . ' . \ . ^ . . . . _ 

ihstructibhal matters. Twelve of the 17 AES items were particularly effective 
in separating the six department heads. 
Revised AES 

Based upon these results and those of a previous study (Hengstier; 
Brandenburg^ Braskamp^^^ and Smbck^ in press) ^ the AES was revised to 11 items 
(Table 55. Eight of the original AES items' were retained, six* were eliminated^ 
two witnessed a slight change in wording and one item was added. 
- ' * The revised AES was completed by faculty (N=87) from four departments in ^ 
Business and Agriculture colleges at University A. Faculty from each 



department ranged from 15 to 36 full-time faculty. A factor- analysis of these 

. _ ' \ _ : ■ . ■ : , _ _ _ 1 

ratings, >^^ing identical procedures described above, again yielded a one factor 

solution. Item correlations with the first principal axis ranged' from .83 (Item 9) 

to .93 (ttem 7). The reliability (coefficient alpha) of tbe revised AES was 

found to be very high (i95)i 

To determine if the revised AES was able to detect differences across 

departments, ratings from the four departments were submitted to a multivariate 

analyses as described above. Significant differences ix =.68, P< .001) were* 



found in the ratings across the four departments. The total discr iminatory 

$ " _ _ / 2 ' 

power of the set of discriminant functions was also very high (a =.88). 

Table 6 presents the standardized discriminant weights for the three 

discriminant functions. Similar patterns were found in the discriminant 

weights with those of the earlier sample form University A. 

. ; DISCUSSION 
The factor analysis of the AES suggests that department administrators are 
evaluated by_ their faculty along one general dimension or factor. This result : 
is consistent with thbs4 obtained in other factor analytic, studies of faculty 



Table 5 



Perfbrmance in handling the routine 
affairs of. the diepartmenti 

Judicious consideration of faculty 
views in dealing vith important 
departmental policies and isjues. 

Effectiveness in providing academic 
direction to the departments 

Effectiveness in communicating 
departmental needs to the Dean and/or 
central administration. 

Effectiveness in establishing and 
implementing budget priorities. 

Leadership in the promotioij and tenure 
process. 

Concern for quality of eduoatidn 
students receive. ^ 

Recognition and reward of faculty 
contributions to the departmert-t. 

Encouragement of fi>search and^ ^ 
scholarly activities among facuLtyi 

Encouragement of good teaching in . 
the department. 

^ : ' ,-- J 

Cdntribiition toward improving/ 
maintaining the reputation of the 
department within the ^discipline. 



Poor 
1 



• 2 

2 
2 

2 
2 
2 

■ 2- 
2 



Excellent 
4 5 * 



no 



Revised-A6S .Standardized DlscrimLriant Wcinhrs. 
for Ratings from Foar University A Departments 



A ES Item 
Old New 

1 1 Performance in handling the routine affairs 

of the department 

2 - 2 Judicious consideration of faculty Ariews in 
• ^dealinr* with important departmental 

policies and issues 



Disc^mii 



5 



: 10 



11 



.12 



18 



Effectiveness in providing academic 
direction to the department. 

Effectiveness in communicating departmental 
needs to the Dean and/or central admini- 
stration. 

Effectiveness in establishing and imple- 
menting budget priorities 



14 10 



11 



Leadership in the promotion and tenure 
process , 

Concern for quality of education students, 
receive .. . ^ 

Recognition and reward of faculty contri- • 
but ions to the department 

Encouragement of research and scholarly 
activities among faculty ^ 

Encburapemeht of good teaching in the 
department - 

Contribution toward improving/ maintaining 
the reputation of the department within • 
the discipline- ^_ 



'^"^epartmcnt 1 

Department 2 
' Department 3 
Department' 4 



.02 ; 



.53 



-i36 



.37 



.74 
.05 
.00 
.75 



it 



-.-3d 



-.12 



.79 



.32 ^i81' 



1.06 



-.48 



-.63 



.11 



.60 



111 



.25 



.44 -1.20 



-.15 



..26 



.31 



.20 



.71 



.88 



-.31 



.08 



.51 -:22 -.54 

Benartmental Ccntrblds 

.72 -1.56 -.37 

2.62 .57 ' .V^ 

-1.55 • -.15 ' lil5 

.1.01 .44 ■ -.58 
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- i-*tattnRS of "the deparemertt hesd .(Span£ier , N6te 5; TbhRer* 1975; Kl^n„^' 
* penham^ 19745 whifch fcxjand a strong factor^ measuring the overall p6rfbrmahce . 
• of the headi . • . i ; ■ 

'The uri id imens tonality of the AES may be;, a function of both a •generosity 
error and a halo 'effect* ^The single factor extracted by Spangler (Note 5) led 
him to conclude that a "halo effect may have obscured the underlying (factor) 
structure" of faculty ' rat in|s of the heads. Faculty may simply have a vested 
interest in the department and_ rt? head and ^ive high ratings to Lac department 
.and its administrator reflecting a potential generosity error.- In a similar 
manner, faculty iay rate the_ administrator in terms of an overall imprd^on 
without differentiating specific aspects of his/her behavior. The potential 
halo effect in the ratinga^s of ten" regarded as invalid variance;, however , 

it could be y4^ed as>.^otential higher order or general factor which docs 

r . . — . , , 

hot necessarily represent invalid variance- 

Another 'likely contributing factor ^ the obsepved uhl-iimehsionality pf . 
the ASS may be the lack of specif icity. in the AES 'items- The items may simply 
be too ganeral in nature to measure specif ic .attributes of the department head. 
This is similar to the argument that is currently being made in regard to, 
. student ratings of instruction (Brandenburg, Derry, a-^.d Hengstler^ Note 8): 



A question* that is often ^ttfed with regard to faculty ratings instruments 
is '.iow effective the instruments are ih detecting clj.-f f erences among. departments 



or 



department heads, in this study, _the AES were found to be very effective in 
discriminating both 'among universities and department administrators. This , 



result is consistent within those obtained by Sunter (note\|95 who found adRnificant 



• * • -■-[■ -- * 

diFf«:finco6 in the responsibilities and characteristics of Idepartment heads from 



sidli and large universities.. Heaas/chairijersbris from .large universities placed 
more importance on personal Research and .scholarship and maintenance of depart- 
jiieHtal records. ' A high^f^mphasis was placed on their research and^^cholar«hip 



•and less on their .teaching experience daring the selection ^roccssi -They ^ 
had less liberty in faculty tenure procedures and 'policy governance^ Heads/ 
chairpersons spent more time serving on standing cbmmit|tees and gradiiate ' 
student advising and less time on teaching, undergraduate student cbhversat ions/ 
advising and student Activities sponsorships They -had more final respon.stbi- 

lities for budget' adihinis^tratidn and dntrdl. One would not expect to find 

^ _ _ _ _ - _ _ _ f 

these s^ine discriminant functions at institutions with different missioi^s^ 

• ' _ ' 

and emphases (eigi small private universities, community colleges) • However, 

one should^ be; able to detect diffareitces in how faculty rate their dc>partment 

head. '* \ " • ' 

In reviewing the results of the AES^discriminant functions and factor 

analysis, one notes that there was little logical correspondence between the 

_ _ __ _ _. 

discriminant fuhctibhs and factors. This apparent discrepancy between dis- 

criminant functions and factors can partially be explained by the purpose of 

the respective statistical procedures.^ in the facto,r analyses, the intent 

was to determine the underlying constructs or dimensions by which faculty rated 

> - ■ ■ - 

the department and administrator^ Items compos irig- a particular factor are^ 

assumed -to be related. In contrast, the discriminant . analysis sought to find 

, ' ' • ' ■ ^; . * ^ , '_ J. \ 

a linear combipation of variables ^itemsT^that showed^ the largest diffeVence 
- - ^ ^ . - M . • : : 

between departmental means. Discriminant analysis makes no assumptions 

regarding, the relationships between items cbmpbsirig a particular function. 

Thus, discriminant and factor analysis have different purpbses in which the 

outcomes are expected to' be diff^itent. * * \ ^ 

^ ^Cft has been argued (Smock ^d Hake, Note 10; Petrie, Note ii; Braslcamp, 

* ■ ■ ' ? * " . ' 

Wise, and Hengstler, 1979; Uhf & Pratt, Ndte*3) that information from a variety of 

. - _ _ _ - . . ^ ' J _ _ / ' 

sources is desirable fbr departmental, and administrator reviews. One source that can 

* - • • . . • • 

' T . ..- '^ - - ;- -_ 

furnish important information, with satisfactory reliability, are faculty ratings. 

- oA 



For sanHi'ative. evaluations of the department, head, faculty ratings can provide 
administrators nith comparative irifbtitiation (iiei discriminant validity). 
For formative evaluations, faculty .ratings can identify relative strengths and 
'weaknesses of a department head; however, the ratings may be affected by a 
halo iffect. Consequently, for formative evaWatioas , more highly detailed . 
xiiagnostic items than the ones included in this study are necessary if, faculty 
^ratings of the head are desired. 
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